Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Constrained reinforcement learning (CRL), also termed as safe learning, is a promising technique enabling the deployment of RL agent in real-world systems. In this paper, we propose successive convex approximation based off-policy optimization (SCAOPO) algorithm to solve general CRL problem, which formulated constrained Markov decision process (CMDP) context average cost. The SCAOPO on solving sequence objective/feasibility problems obtained by replacing objective and constraint functions original problem with surrogate functions. proposed enables reuse experiences from previous updates, thereby significantly reducing implementation cost when deployed engineering systems that need online learn environment. spite time-varying state distribution stochastic bias incurred feasible initial point can still provably converge Karush-Kuhn-Tucker (KKT) almost surely.